Systems and methods for utilizing artificial intelligence to guide a medical device

ABSTRACT

Systems and methods for generating navigational guidance for a medical device within a body are disclosed. One computer-implemented method may include: receiving, at a computer server, image data associated with at least one anatomical object; determining, using a processor associated with the computer server and via application of a trained predictive navigational guidance model to the image data, navigational guidance for the medical device in relation to the at least one anatomical object; generating, based on the determining, at least one visual representation associated with the navigational guidance; and transmitting, to a user device in network communication with the computer server, instructions to display the at least one visual representation associated with the navigational guidance overtop of the image data on a display screen of the user device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication No. 63/368,529, filed Jul. 15, 2022, which is incorporatedby reference herein in its entirety.

TECHNICAL FIELD

Various aspects of this disclosure relate generally to systems andmethods for utilizing artificial intelligence to provide navigationalguidance for a medical device performing actions within a body. Morespecifically, in embodiments, this disclosure relates to the applicationof a trained machine learning model to data associated with providingpredictive navigational guidance for a physician operating a medicaldevice.

BACKGROUND

Certain medical procedures may be performed to examine and treat issuesinternal to the body. For example, during an endoscopic procedure, along, thin tube is inserted directly into the body to observe aninternal organ or tissue in detail. Such a procedure may also be used tocarry out other tasks, including imaging and minor surgery. In someendoscopic procedures, cannulation of various anatomical objects (e.g.,one or more ducts, etc.) may need to be achieved via insertion of anendoscopic component (e.g., a guidewire). Such a maneuver may be verychallenging and may carry with it a steep learning curve. Consequently,the time for a novice physician to become proficient with such aprocedure may be very long.

This disclosure is directed to addressing above-referenced challenges.The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY

Each of the aspects disclosed herein may include one or more of thefeatures described in connection with any of the other disclosedaspects.

Aspects of the disclosure relate to, among other things, systems andmethods for generating navigational guidance for a medical deviceoperating within a body. According to an example, a computer-implementedmethod is provided for generating navigational guidance for a medicaldevice within a body. The computer-implemented method, including:receiving, at a computer server, image data associated with at least oneanatomical object; determining, using a processor associated with thecomputer server and via application of a trained predictive navigationalguidance model to the image data, navigational guidance for the medicaldevice in relation to the at least one anatomical object; generating,based on the determining, at least one visual representation associatedwith the navigational guidance; and transmitting, to a user device innetwork communication with the computer server, instructions to displaythe at least one visual representation associated with the navigationalguidance overtop of the image data on a display screen of the userdevice.

Any of the computer-implemented methods for generating navigationalguidance may include any of the following features and/or processes. Themedical device may be an endoscope having an extendable guide wire. Theat least one anatomical objet may correspond to one or more of: apapilla, an orifice, and/or an internal duct. The navigational guidancemay include a path for the medical device for cannulation of theanatomical object. The image data may be captured by at least one sensorassociated with the medical device and/or by at least one other imagingdevice. The at least one sensor may contain a camera sensor and imagedata captured by the camera sensor may include at least one: shape data,orientation data, and/or appearance data of the at least one anatomicalobject. The at least one other imaging device may contain an X-raydevice and/or an ultrasound device and the image data captured by the atleast one other imaging device may include anatomical structure data.One or more other sensors may be utilized, including at least one of: anelectromagnetic sensor, an accelerometer, a gyroscope, a fiber opticsensor, an ultrasound transducer, a capacitive position sensor, and/oran inductive position sensor. The one or more other sensors may captureposition data associated with the medical device. The determination ofthe navigational guidance may include identifying anatomical featuredata from the image data using the predictive navigational guidancemodel. The identification of the anatomical feature data may include:identifying a first classification associated with a first anatomicalobject within a first target region of the image data; identifying asecond classification associated with a second anatomical object fromwithin a second target region bounded by the first target region;detecting a location of one or more third anatomical objects from withinthe second target region; and detecting one or more other anatomicalobjects associated with the first anatomical object. The determinationof the navigational guidance for the medical device may include:identifying a confidence weight held by the predictive navigationalguidance model for the at least one anatomical object; and determiningwhether that confidence weight is greater than a predeterminedconfidence threshold; wherein the generation of the navigationalguidance is only performed in response to determining that theconfidence weight is greater than the predetermined confidencethreshold. The at least one visual representation may include one ormore of: at least one trajectory overlay, at least one annotation,and/or at least one feedback notification. The at least one trajectoryoverlay may include a visual indication, overlaid on top of an image ofthe at least one anatomical object, of a projected path to an accesspoint of the at least one anatomical object that a component of themedical device may follow to cannulate the at least one anatomicalobject. The computer-implemented method may also receive position datafor the medical device and identify deviation of the medical device fromthe projected path based on analysis of the position data. Thegeneration of the feedback notification in this situation may beresponsive to the detection that the deviation of the medical devicefrom the projected path is greater than a predetermined amount. The atleast one annotation may include one or more visual indications,overlaid on top of an image of the at least one anatomical object,indicating predetermined features associated with the at least oneanatomical object. The one or more visual indications may include one ormore of: a color indication, an outline indication, and/or a text-basedindication.

According to another example, a computer-implemented method of traininga predictive navigational guidance model is provided. Thecomputer-implemented method, including: receiving, from a database, atraining dataset comprising historical medical procedure data associatedwith a plurality of completed medical procedures; extracting, from imagedata in the training dataset, anatomical feature data; extracting, fromsensor data in the training dataset, medical device positioning data;extracting, from the training dataset, procedure outcome data; andutilizing the extracted anatomical feature data, the extracted medicaldevice positioning data, and the extracted procedure outcome data totrain the predictive navigational guidance model.

Any of the computer-implemented methods for training a predictivenavigational guidance model may include any of the following featuresand/or processes. The training dataset may be annotated withidentification data. The extraction of the anatomical feature data mayinclude: identifying a classification associated with a first anatomicalobject; determining an identity of a second anatomical object fromwithin a second target region bounded by the first target region in theimage data; detecting at least one location associated with one or morethird anatomical objects; and detecting one or more other anatomicalobjects associated with the first anatomical object. Thecomputer-implemented method may also identify a new procedural outcomeand update the database with data associated with the new proceduraloutcome.

According to another example, a computer system for generatingnavigational guidance for a medical device within a body, the computersystem includes: at least one memory storing instructions; at least oneprocessor configured to execute the instructions to perform operationscomprising: receiving image data associated with at least one anatomicalobject; determining, using the at least one processor and viaapplication of a trained predictive navigational guidance model to theimage data, navigational guidance for the medical device in relation tothe at least one anatomical object; generating, based on thedetermining, at least one visual representation associated with thenavigational guidance; and transmitting, to a user device, instructionsto display the at least one visual representation associated with thenavigational guidance overtop of the image data on a display screen ofthe user device

It may be understood that both the foregoing general description and thefollowing detailed description are exemplary and explanatory only andare not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate aspects of this disclosure andtogether with the description, serve to explain the principles of thedisclosure.

FIG. 1 depicts an exemplary environment for training and/or utilizing amachine-learning model to apply to medical procedure data to generatepredictive navigational guidance, according to one or more embodiments.

FIG. 2 depicts a flowchart of an exemplary method of training amachine-learning model to provide predictive navigational guidanceduring a medical procedure, according to one or more embodiments.

FIG. 3 depicts a flowchart of an exemplary method of training amachine-learning model to extract anatomical feature data, according toone or more embodiments.

FIG. 4 depicts a plurality of views of different types of an anatomicalobject, according to one or more embodiments.

FIG. 5 depicts a plurality of views of different characteristic patternsof an anatomical object, according to one or more embodiments.

FIG. 6 depicts a plurality of views of anatomical object differentiationbased on the characteristic patterns of an anatomical object, accordingto one or more embodiments.

FIG. 7 depicts a flowchart of an exemplary method of providingpredictive navigational guidance during a medical procedure using atrained machine-learning model, according to one or more embodiments.

FIG. 8 depicts an illustration of annotated-based predictive guidance,according to one or more embodiments.

FIG. 9 depicts an illustration of trajectory-based predictive guidance,according to one or more embodiments.

FIG. 10 depicts an example of a computing device, according to one ormore embodiments.

It is to be that both the foregoing general description and thefollowing detailed description are exemplary and explanatory only andare not restrictive of the disclosed embodiments, as claimed.

DETAILED DESCRIPTION

The terminology used below may be interpreted in its broadest reasonablemanner, even though it is being used in conjunction with a detaileddescription of certain specific examples of the present disclosure.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection. Both the foregoing general description and the followingdetailed description are exemplary and explanatory only and are notrestrictive of the features, as claimed.

In this disclosure, the term “based on” means “based at least in parton.” The singular forms “a,” “an,” and “the” include plural referentsunless the context dictates otherwise. The term “exemplary” is used inthe sense of “example” rather than “ideal.” The terms “comprises,”“comprising,” “includes,” “including,” or other variations thereof, areintended to cover a non-exclusive inclusion such that a process, method,or product that comprises a list of elements does not necessarilyinclude only those elements, but may include other elements notexpressly listed or inherent to such a process, method, article, orapparatus. The term “diameter” may refer to a width where an element isnot circular. The term “top” refers to a direction or side of a devicerelative to its orientation during use, and the term “bottom” refers toa direction or side of a device relative to its orientation during usethat is opposite of the “top.” The term “exemplary” is used in the senseof “example,” rather than “ideal.” Relative terms, such as,“substantially” and “generally,” are used to indicate a possiblevariation of ±10% of a stated or understood value.

Reference to any particular procedure is provided in this disclosureonly for convenience and not intended to limit the disclosure. A personof ordinary skill in the art would recognize that the conceptsunderlying the disclosed devices and methods may be utilized in anysuitable procedure. For ease of description, portions of the deviceand/or its components are referred to as proximal and distal portions.It should be noted that the term “proximal” is intended to refer toportions closer to a user of the device, and the term “distal” is usedherein to refer to portions further away from the user. Similarly,extends “distally” indicates that a component extends in a distaldirection, and extends “proximally” indicates that a component extendsin a proximal direction.

In the following description, embodiments will be described withreference to the accompanying drawings. As will be discussed in moredetail below, according to certain aspects of the disclosure, methodsand systems are disclosed for capturing information associated with oneor more biological components using a medical device (e.g., during amedical procedure), comparing the captured information against adatabase of historical procedural data or applying a model trained onhistorical procedural data to the captured information, and thereafterproviding various types of guidance based on the results of thecomparison and/or analysis.

Endoscopic Retrograde Choloangio-Panceatography (ERCP) is a procedureconventionally utilized to examine the biliary duct. In the procedure,an endoscope is inserted through the mouth and is passed to theduodenum. The duodenum is then insufflated and the entry point for thecommon duct for the biliary and pancreatic ducts is identified. A tomemay be used to perform a sphincterotomy to widen the opening, therebymaking cannulation easier to perform. A guidewide may then be used toenter into the common duct, and is maneuvered to the biliary duct. Onceduct cannulation has been achieved, a cholangioscope may be insertedover the guidewire and into the duct. Contrast may then be injected andused in combination with X-rays to identify regions of interest. Thephysician may thereafter perform a variety of procedures such as stonemanagement or therapy of biliary malignancies.

Conventionally, cannulation of the proper duct can be very challengingfor a variety of reasons. For example, the ergonomics of manipulating an8-degree-of-freedom endoscope to enter into a precise location can bedifficult, even for an experienced and practiced physician.Additionally, as another example, the lack of visualization of the ductpathway beyond the common entry point may exacerbate the difficulty ofthe task. More particularly, although various types of visualizations ofa target area are available to a physician during the ERCP procedure(e.g., pre-operative magnetic resonance cholangio-pancreatography(MRCP), post-cannulation high resolution imaging, X-rays, pre-operativeCT scans, etc.), only direct visualization is utilized (i.e., asprovided by the endoscope) for the cannulation process specifically.This limited visualization provides no information regarding the anatomyof the ducts beyond the common entry point, which is especiallyproblematic because the anatomical architecture of the ducts is patientspecific (i.e., the characteristics of every single papilla throughwhich the guidewire needs to enter is different). Consequently, in viewof the aforementioned challenges, a common procedural result isdisturbance of the pancreatic duct (e.g., via misplacement of theguidewire by the physician, etc.). In more serious cases, thisdisturbance may lead to pancreatitis.

The high degree of inherent difficulty operating the endoscope, coupledwith the lack of proper visualization during cannulation, results in asteep learning curve for physicians attempting to attain proficiency inconducting ERCP procedures. Furthermore, even after becoming proficient,physicians need to continually perform these types of procedures tomaintain their level of skill (e.g., at least one ERCP procedure aweek), which may be very demanding, burdensome, and/or not feasible(e.g., a physician may not be located in an area where a high volume ofERCP procedures are performed, etc.). Accordingly, a need exists for theERCP procedure to be simplified, or modified, to enable more physiciansto master the procedure in a shorter period of time, which maypotentially lead to better patient care.

As will be discussed in more detail below, the present disclosureprovides a platform that may provide dynamic guidance to a physicianduring a procedure such as an ERCP procedure by applying a predictivenavigational guidance model (i.e., trained from historicalprocedure-related data stored in an accessible ERCP database) to dataobtained and associated with a live medical procedure. Moreparticularly, anatomical feature data (e.g., characteristics of a targetpapilla and/or the anatomy of the relevant ducts) may be extracted fromimage data initially captured using one or more sensors (e.g.,camera/video sensors, etc.) associated with an endoscope and/or one ormore other imaging modalities (e.g., fluoroscopy, ultrasound, etc.).Additionally, in some embodiments, medical device position data (e.g.,the position, angle, and/or movements of a medical device with respectto a target anatomical object) may be captured using one or more othersensors (e.g., electromagnetic sensors, etc.). The accumulated liveprocedure data may then be submitted as input to the predictivenavigational guidance model, which may then analyze the data todetermine navigational guidance for maneuvering of a guidewire of anendoscope through an appropriate orifice of a papilla. This guidance maybe transmitted to a user device (e.g., a computing device integrally oroperatively coupled to the endoscope, etc.) and may manifest as one ormore visual indications (e.g., recommended cannulation trajectories,annotations, notifications, etc.) that may be overlaid atop the liveprocedural image data to aid physicians in the completion of theprocedure.

It is important to note that although the techniques utilized herein aredescribed with explicit reference to an ERCP procedure, such adesignation is not limiting. More particularly, the machine-learningmodel described herein may be trained to identify characteristicfeatures associated with other anatomical objects/structures and maycorrespondingly be utilized to provide guidance for other types ofmedical procedures.

FIG. 1 depicts an exemplary environment 100 that may be utilized withthe techniques presented herein. One or more user device(s) 105 maycommunicate with one or more medical devices 110 and/or one or moreserver system(s) 115 across a network 101. The one or more userdevice(s) 105 may be associated with a user, e.g., a user associatedwith one or more of generating, training, using, or tuning amachine-learning model for providing predictive navigational guidanceduring a medical procedure. For example, the one or more user device(s)105 may be associated with a physician performing a medical procedure,e.g., an ERCP procedure, and seeking to gain the benefits derived fromthe capabilities of the server system(s) 115.

In some embodiments, the components of the environment 100 may beassociated with a common entity, e.g., a single business ororganization, or, alternatively, one or more of the components may beassociated with a different entity than another. The systems and devicesof the environment 100 may communicate in any arrangement. For example,one or more user device(s) 105 and/or medical devices 110 may beassociated with one or more clients or service subscribers, and serversystem(s) 115 may be associated with a service provider responsible forreceiving procedural data from the one or more clients or servicesubscribers and thereafter utilizing the capabilities of the serversystem(s) 115 to return an output to the one or more clients or servicesubscribers. As will be discussed further herein, systems and/or devicesof the environment 100 may communicate in order to generate, train,and/or utilize a machine-learning model to characterize aspects of amedical procedure and dynamically provide predictive navigationalguidance, among other activities.

The user device 105 may be configured to enable the user to accessand/or interact with other systems in the environment 100. For example,the user device 105 may be a computer system such as, for example, adesktop computer, a mobile device, a tablet, etc. In some embodiments,the user device 105 may include one or more electronic application(s),e.g., a program, plugin, browser extension, etc., installed on a memoryof the user device 105.

The user device 105 may include a display user interface (UI) 105A, aprocessor 105B, a memory 105C, and a network interface 105D. The userdevice 105 may execute, by the processor 105B, an operating system (O/S)and at least one electronic application (each stored in memory 105C).The electronic application may be a desktop program, a browser program,a web client, or a mobile application program (which may also be abrowser program in a mobile O/S), an applicant specific program, systemcontrol software, system monitoring software, software developmenttools, or the like. For example, environment 100 may extend informationon a web client that may be accessed through a web browser. In someembodiments, the electronic application(s) may be associated with one ormore of the other components in the environment 100. The application maymanage the memory 105C, such as a database, to transmit medicalprocedure data to network 101. The display/UI 105A may be a touch screenor a display with other input systems (e.g., mouse, keyboard, etc.) sothat the user(s) may interact with the application and/or the O/S. Thenetwork interface 105D may be a TCP/IP network interface for, e.g.,Ethernet or wireless communications with the network 110. The processor105B, while executing the application, may generate data and/or receiveuser inputs from the display/UI 105A and/or receive/transmit messages tothe server system 115, and may further perform one or more operationsprior to providing an output to the network 110.

The medical device(s) 110 in the environment 100 may include one or moremedical devices (e.g., an endoscope, other internal imaging devices,etc.) integrally (e.g., via a wired connection, etc.) or operatively(e.g., via a wireless connection, etc.) coupled to the user device(s)105 and/or the server system 115. Data obtained by sensors of themedical device(s) 110 (e.g., image/video data, position data, etc.) maybe transmitted to one or both of the user device 105 and/or the serversystem 115.

In various embodiments, the network 101 may be a wide area network(“WAN”), a local area network (“LAN”), a personal area network (“PAN”),or the like. In some embodiments, network 101 includes the Internet, andinformation and data provided between various systems occurs online.“Online” may mean connecting to or accessing source data or informationfrom a location remote from other devices or networks coupled to theInternet. Alternatively, “online” may refer to connecting or accessing anetwork (wired or wireless) via a mobile communications network ordevice. The Internet is a worldwide system of computer networks—anetwork of networks in which a party at one computer or other deviceconnected to the network can obtain information from any other computerand communicate with parties of other computers or devices. The mostwidely used part of the Internet is the World Wide Web(often-abbreviated “WWW” or called “the Web”). A “website page”generally encompasses a location, data store, or the like that is, forexample, hosted and/or operated by a computer system so as to beaccessible online, and that may include data configured to cause aprogram such as a web browser to perform operations such as send,receive, or process data, generate a visual display and/or aninteractive interface, or the like.

The server system 115 may include an electronic data system,computer-readable memory such as a hard drive, flash drive, disk, etc.In some embodiments, the server system 115 includes and/or interactswith an application programming interface for exchanging data to othersystems, e.g., one or more of the other components of the environment100. The server system 115 may include and/or act as a repository orsource for extracted raw dataset information.

The server system 115 may include a database 115A and at least oneserver 115B. The server system 115 may be a computer, system ofcomputers (e.g., rack server(s)), and/or or a cloud service computersystem. The server system may store or have access to database 115A(e.g., hosted on a third party server or in memory 115E). The server(s)may include a display/UI 115C, a processor 115D, a memory 115E, and/or anetwork interface 115F. The display/UI 115C may be a touch screen or adisplay with other input systems (e.g., mouse, keyboard, etc.) for anoperator of the server 115B to control the functions of the server 115B.The server system 115 may execute, by the processor 115D, an operatingsystem (O/S) and at least one instance of a servlet program (each storedin memory 115E). When user device 105 or medical device 110 sendsmedical procedure data to the server system 115, the received datasetand/or dataset information may be stored in memory 115E or database115A. The network interface 115F may be a TCP/IP network interface for,e.g., Ethernet or wireless communications with the network 101.

The processor 115D may include and/or execute instructions to implementa predictive navigational guidance platform 120, which may include amedical procedure database 120A (e.g., containing data associated withhistorical ERCP procedures, etc.) and/pr a navigational guidance model120B. The medical procedure database 120A may be continually updated(e.g., with new medical procedure data). Additionally, the medicalprocedure database 120A may also be utilized to train the navigationalguidance model 120B to dynamically identify, from data associated withan instant medical procedure, correlations between the characteristicsassociated with certain anatomical objects, the positioning of one ormore components of a medical device, and/or the corresponding outcome ofthe procedure. The process by which these correlations may be identifiedis later described herein by the disclosure associated with FIG. 3 .

In an embodiment, the medical procedure database 120A and thenavigational guidance model 120B may both be contained within thepredictive navigational guidance platform 120. Alternatively, one orboth of these components may be subcomponents of other components withineach other or may be resident on other components of the environment100. For example, the medical procedure database 120A may beincorporated into an application platform on the user device 205 whereasthe navigational guidance model 120B may be resident on the server 115Bof the server system 115.

As discussed in further details below, the server system 115 maygenerate, store, train, or use one or more machine-learning modelsconfigured to analyze medical procedure data and provide predictivenavigational guidance based on that analysis. The server system 115 mayinclude one or more machine-learning models and/or instructionsassociated with each of the one or more machine-learning models, e.g.,instructions for generating a machine-learning model, training themachine-learning model, using the machine-learning model, etc. Theserver system 115 may include instructions for retrieving outputfeatures, e.g., based on the output of the machine-learning model,and/or operating the displays 105A and/or 115C to generate one or moreoutput features, e.g., as adjusted based on the machine-learning model.

The server system 115 may include one or more sets of training data. Thetraining data may contain various types of historical data regarding aspecific medical procedure, such as ERCP. For example, the training datamay include characteristic information associated with various types ofdetected papilla (e.g., shape data, size data, orientation data,appearance data, etc.), orifice characteristic information (e.g.,number, size, location, duct-association, etc.) associated with each ofthe detected papilla, anatomical information associated with thelocation and/or structure of a biliary and/or pancreatic duct,additional anatomic feature information associated with the detectedpapilla (e.g., presence or absence of intramural folds, oralprotrusions, frenulum and/or sulcus, etc.), position of an endoscopewith respect to the papilla before and/or during cannulation, historicalERCP procedure outcomes, and the like.

In some embodiments, a system or device other than the server system 115may be used to generate and/or train the machine-learning model. Forexample, such a system may include instructions for generating themachine-learning model, the training data and ground truth, and/orinstructions for training the machine-learning model. A resultingtrained machine-learning model may then be provided to the server system115.

In some embodiments, a machine-leaning model based on neural networksincludes a set of variables, e.g., nodes, neurons, filters, etc., thatare tuned, e.g., weighted or biased, to different values via theapplication of training data. In other embodiments, a machine learningmodel may be based on architectures such as support-vector machines,decision trees, random forests or Gradient Boosting Machines (GBMs).Alternate embodiments include using techniques such as transferlearning, wherein one or more pre-trained machine learning models onlarge common or domain specific dataset may be leveraged for analyzingthe training data.

In supervised learning, e.g., where a ground truth is known for thetraining data provided, training may proceed by feeding a sample oftraining data into a model with variables set at initialized values,e.g., at random, based on Gaussian noise, a pre-trained model, or thelike. The output may be compared with the ground truth to determine anerror, which may then be back-propagated through the model to adjust thevalues of the variable.

Training may be conducted in any suitable manner, e.g., in batches, andmay include any suitable training methodology, e.g., stochastic ornon-stochastic gradient descent, gradient boosting, random forest, etc.In some embodiments, a portion of the training data may be withheldduring training and/or used to validate the trained machine-learningmodel, e.g., compare the output of the trained model with the groundtruth for that portion of the training data to evaluate an accuracy ofthe trained model. The training of the machine-learning model may beconfigured to cause the machine-learning model to learn contextualassociations between the raw procedure data and the context with whichit is associated with (e.g., which anatomical features and/or medicaldevice actions affected the success rate of the ERCP procedure etc.),such that the trained machine-learning model is configured to providepredictive guidance that may increase the success rate of an ERCPprocedure.

In various embodiments, the variables of a machine-learning model may beinterrelated in any suitable arrangement in order to generate theoutput. For instance, in some embodiments, the machine-learning modelmay include signal processing architecture that is configured toidentify, isolate, and/or extract features, patterns, and/or structurein an image or video. For example, the machine-learning model mayinclude one or more convolutional neural networks (“CNN”) configured toidentify anatomical features associated with a papilla and relatedanatomical structures and may include further architecture, e.g., aconnected layer, neural network, etc., configured to determine arelationship between the identified features and structures in order todetermine an optimal cannulation path.

For example, in some embodiments, the machine-learning model of theserver system 115 may include a Recurrent Neural Network (“RNN”).Generally, RNNs are a class of feed-forward neural networks that may bewell adapted to processing a sequence of inputs. In some embodiments,the machine-learning model may include a Long Short Term Memory (“LSTM”)model and/or Sequence to Sequence (“Seq2Seq”) model. An LSTM model maybe configured to generate an output from a sample that takes at leastsome previous samples and/or outputs into account. A Seq2Seq model maybe configured to, for example, receive a sequence of images as input andthereafter generate a sequence of annotations and/or predictive medicaldevice movement trajectories as output.

Although depicted as separate components in FIG. 1 , a component orportion of a component in the environment 100 may, in some embodiments,be integrated with or incorporated into one or more other components.For example, a portion of the display 115C may be integrated into theuser device 105 or the like. In some embodiments, operations or aspectsof one or more of the components discussed above may be distributedamongst one or more other components. Any suitable arrangement and/orintegration of the various systems and devices of the environment 100may be used.

Training a Machine-Learning Model to Provide Predictive NavigationalGuidance

FIG. 2 illustrates an exemplary process for training a machine-learningmodel, such as a navigational guidance model 120B, to identify keyanatomical features during an ERCP procedure and to provide dynamicguidance to aid an operating physician.

At step 205 of the training process, the method may include receiving atraining dataset, e.g., a compilation of data associated with previouslycompleted ERCP procedures. More particularly, for each completed ERCPprocedure, the training data may include images, videos, medicalreports, etc. associated with one or more anatomical objects of interestdetected during previously completed ERCP procedures (e.g., a papilla,one or more orifices on the papilla, a biliary duct, a pancreatic ductetc.). This data may have been captured using one or more sensorsassociated with the medical device (e.g., an optical camera) and/orother imaging modalities (X-ray imaging, fluoroscopy, etc.). In anembodiment, the training data may also include position and/movementdata of a medical device (e.g., an endoscope) and/or components thereof(e.g., a guidewire) in relation to one or more anatomical objects duringthe procedure. The position and/or movement data may have been capturedusing one or more other sensors (e.g., electromagnetic (EM) sensors,accelerometers, gyroscopes, fiber optics, ultrasound transducer,capacitive or inductive position sensors, etc.), and/or may have beenobtained via any other suitable means, e.g., via observation by a personand/or automated system, via feedback of a controller for the medicaldevice, etc. In an embodiment, the training data may also contain anindication of the outcome of each of the completed ERCP procedures(e.g., positive outcome, negative outcome, severity of negative outcome,etc.).

In an embodiment, each article of training data may be pre-annotatedwith relevant anatomical feature information. For example, each image ofa papilla may identify one or more of: a classification associated withthe papilla based on shape, orientation, and/or appearance data;orifice(s) on the papilla and their corresponding association (e.g.,orifice associated with pancreatic duct, orifice associated with biliaryduct, etc.); the orientation of each orifice (e.g., down-facing,side-facing, etc.); intramural ligament features (e.g., intramuralfolds), diverticulum, oral protrusions; frenulum and/or sulcus; and thelike. In an embodiment, each article of training data may identify apath followed by a medical device to cannulate one or both of theorifices.

In an embodiment, a server system (e.g., server system 115) may receivethe training dataset and may store the training dataset in a database(e.g., ERCP database 120A on the predictive navigational guidanceplatform 120, the database 115A, etc.) and/or in a memory (e.g., memory115E). In an embodiment, a user may upload the training dataset to auser device (e.g., user device 105) to manually annotate each article oftraining data. The user device 105 may or may not store the trainingdataset in the memory (e.g., 105C). Once annotated, the user device 105may transmit the annotated training dataset to the server system 115 viaa network 101.

At step 210, the method may include, for each training datasetassociated with an ERCP procedure, extracting anatomical feature datafrom the annotated training data. The extracted anatomical feature datamay be used to train the machine-learning model to correctly identifyand differentiate, during a live procedure, important anatomical objectsrelevant to the ERCP procedure. Additional disclosure relating to howthe machine-learning model is trained off of the extracted anatomicalfeature data is further provided below in the discussion of FIG. 3 .

Turning back to FIG. 2 , at step 215, positional data associated withthe recorded position, angle, and/or movements of an endoscope (andcomponents thereof) with respect to a papilla during cannulation in theERCP procedures in the training dataset may be extracted. Thispositional data may be originally acquired using one or more integrallyor operatively coupled sensors to the endoscope and/or guidewireincluding, but not limited to, electromagnetic (EM) sensors,accelerometers, gyroscopes, fiber optics, ultrasound transducer,capacitive or inductive position sensors, etc.). In an embodiment, thepositional data may be annotated with positive and/or negativeannotations. For example, positional data that led to a successfulprocedure (e.g., resulting from an optimal position and/or angle of theendoscope with respect to the papilla, etc.) may be classified aspositive whereas positional data that led to an unsuccessful procedure(e.g., due to an angle of approach, etc.) may be classified as negative.Accordingly, the predictive navigational guidance model may train on theaccumulation of the positional data to identify the ideal positions,angles, and/or movements of the endoscope and guidewire to successfullycannulate the papilla.

At step 220, outcome data associated with the ERCP procedures in thetraining dataset may be extracted and utilized to train the predictivenavigational guidance model. More particularly, each of the ERCPprocedures in the training dataset may contain one or more indicationsof how successful or unsuccessful the ERCP procedure was. For thetraining dataset, the outcome data for each ERCP procedure may beexplicitly annotated so that the predictive navigational guidance modelmay learn to dynamically distinguish between successful proceduresand/or unsuccessful procedures and sub-steps thereof.

In an embodiment, the outcome data may provide a binary indication ofthe success state of the ERCP procedure (e.g., the ERCP procedure wasoverall successful or unsuccessful, etc.). In this regard, the successstate of the ERCP procedure may be based on whether or not the biliaryduct was successfully cannulated. Additionally or alternatively, inanother embodiment, the outcome data may provide more granularindications of procedural outcomes occurring during the course of theERCP procedure. More particularly, the outcome data may delineate theportions of the ERCP procedure that were successful (e.g., the biliaryorifice and pancreatic orifice were successfully distinguished from oneanother) and unsuccessful (e.g., cannulation of the biliary duct wasunsuccessful due to the approach angle of the guidewire, etc.).

At step 225, the accumulation of all the extracted data from steps210-220 may be utilized to train the predictive navigational guidancemodel. In this regard, the trained predictive navigational guidancemodel may thereafter be able to receive data associated with a live ERCPprocedure and apply the knowledge obtained from the training procedureto identify correlations between aspects of the live ERCP procedure andaspects associated with previously completed ERCP procedures (e.g.,those embodied in the trained dataset, etc.). Thereafter, the predictivenavigational guidance model may be able to provide dynamic guidance toan operator of the medical device (e.g., a physician, etc.), as furtherdescribed herein.

FIG. 3 illustrates an exemplary process of extracting anatomical featuredata from an annotated training dataset. More particularly, acombination of various visual AI neural network frameworks may beutilized to isolate and evaluate progressively smaller and/or morespecific areas of a target image based on ROI and specific goals. Oncetrained, the predictive navigational guidance model 120B may be able todynamically and accurately identify specific objects and ROIs fromreceived data associated with a live ERCP procedure.

At step 305, the method may first include training the predictivenavigational guidance model to classify a papilla type. Such aclassification is important because the type of papilla present maydictate the location, orientation, and/or structure of other anatomicalobjects (e.g., orifices, etc.). Possible papilla types include: regular,small, protruding or pendulous, and creased or ridged. For example, withreference to FIG. 4 , a plurality of potential papilla types areillustrated. For instance, with respect to papilla 405, ROI 405Aincludes a “regular” papilla 405B, i.e., one that contains nodistinctive features and has a “classic” appearance. With respect topapilla 410, ROi 410A includes a “small” papilla 410B, i.e., one that isoften flat, with a diameter less than or equal to 3 millimeters. Withrespect to papilla 415, ROI 415A may include a protruding or pendulouspapilla 415B, i.e., one that may stand out, protrude, or bulge into theduodenal lumen or sometimes hang down and may be pendulous in appearancewith the orifice oriented caudally. With respect to papilla 420, ROI420A may include a creased or ridged papilla 420B, i.e., one in whichthe ductal mucosa appears to extend distally out of the papillaryorifice either on a ridge or in a crease.

In an embodiment, each set of annotated training data may contain anexplicit designation of the ROI as well as an explicit indication of thepapilla type. In an embodiment, each image in the training set maygenerally be captured with an “en face” alignment to the target papilla.In an embodiment, for step 305, a regional convolution neural network(R-CNN) framework, e.g., RESNET-18, may be employed and trained on ahigh batch volume of annotated images of papilla types to facilitateproper papilla type classification during a live ERCP procedure.

At step 310, the method may include training the predictive navigationalguidance model to identify an orifice type. More particularly, in anembodiment, the predictive navigational guidance model may be trained toidentify a second ROI (i.e., ROI-2), bounded by the first ROI (i.e.,ROI-1), which is associated with a particular papilla pattern. The typeof papilla pattern identified may correspondingly dictate thecharacteristics of one or more orifices resident on the papilla. Turningnow to FIG. 5 , a plurality of known papilla patterns are illustrated.For example, Papilla-A 505 may correspond to a typical papilla patternwith an annular shape, with some having nodular changes at the oral sideof the center. Papilla-U 510 may correspond to a papilla pattern thatmay be generally unstructured without a clear orifice. Papilla-LO 515may correspond to a papilla pattern having longitudinal groovescontinuous with the orifice, with the length of the grooves being longerthan a transverse diameter of the biliary duct axis of the papilla.Papilla-I 520 may correspond to a papilla pattern have two separate,isolated orifices of the biliary and pancreatic ducts, with the openingon the oral side being that of the biliary duct and that on the analside being that of the pancreatic duct. Papilla-G 525 may correspond toa papilla pattern having a gyrate structure.

In an embodiment, each set of annotated training data may contain one ormore explicit designations that identify: the type of papilla patternexpressed by the papilla, the region on the papilla where one or moreorifices are located based on the papilla pattern, and the type ofaccess the orifice may present. In an embodiment, as a result of thesmaller focus area of ROI-2 compared to ROI-1, fewer convolutions may beneeded to accurately classify the orifice type. Accordingly, for step310, a Fast R-CNN framework, e.g., RESTNET-9 or other fasterconventional R-CNNs, may be employed and trained on a high batch volumeof annotated papilla patterns and corresponding orifice characteristics.

At step 315, the method may include training the predictive navigationalguidance model to detect the location of the biliary and/or pancreaticducts. More particularly, in an embodiment, the biliary and/orpancreatic ducts may be located on the image and, if possible,distinguished from each other and/or other anatomical objects (e.g.,based on annotations in the training data). Turning now to FIG. 6 ,non-limiting examples are provided of situations where the biliaryand/or pancreatic ducts may be explicitly delineated on each of theimages from FIG. 5 . For example, Papilla 605 illustrates locations ofboth the biliary duct 605A and pancreatic duct 605B, Papilla 610illustrates the location of the biliary duct 610A, Papilla 615illustrates the location of the biliary duct 615A, Papilla 620illustrates the locations of both the biliary duct 620A and pancreaticduct 620B, and Papilla 625 illustrates the location of the biliary duct625A.

In an embodiment, duct differentiation may be accomplished using asemantic segmentation model (e.g., SegNet) that may employ a fullconvolutional network on just the region bounded by ROI-2. Such a modelmay utilize a two stage approach to first distinguish the ducts from thesurrounding anatomical features found on the papilla and thereafter mayperform regression to differentiate the ducts from one another. In anembodiment, the full convolutional network may be trained using a highvolume of annotated images delineating the identity of the biliaryand/or pancreatic ducts.

At steps 320-330, one or more detection algorithms may be leveraged toidentify specific anatomical features on, or associated with, thepapilla. For example, at step 320, a detection algorithm may be trainedto determine if any intramural folds exist. At step 325, subsequent to,or independent from, the detection training process performed at step320, the same or different detection algorithm may be trained todetermine if oral protrusions exist. At step 330, subsequent to, orindependent from, the detection process performed at steps 320 and 325,the same or different detection algorithm may be trained to determine ifa frenulum and/or a sulcus are present. To train the detectionalgorithms at each of steps 320-330 to be primed to detect the specificanatomical feature(s) associated with each step, the training datasetmay be annotated with the relevant anatomical objects.

Applying the Trained Machine-Learning Model to Live Procedure Data toGenerate Predictive Navigational Guidance

FIG. 7 illustrates an exemplary process for determining predictivenavigational guidance for an ERCP procedure and thereafter providing theguidance to an operating physician.

At step 705, an embodiment of the trained predictive navigational modelmay receive image data associated with one or more anatomical objectsassociated with a live ERCP procedure. For example, the one or moreanatomical objects may correspond to a papilla, one or more orificesresident on the papilla, a biliary and/or pancreatic duct, otheranatomical features or structures associated with any of the foregoing,and the like. In an embodiment, the image data may be captured by one ormore optical sensors of a medical device utilized in the ERCP procedure.For example, the image data may be captured by one or more opticalsensors positioned on a distal end of an endoscope.

At step 710, an embodiment of the trained predictive navigationalguidance model may determine predictive navigational guidance for themedical device utilized in the medical procedure in relation to a targetanatomical object. In this regard, an embodiment may apply the imagedata as input to a trained predictive navigational guidance model. Thetrained predictive navigational guidance model may be configured toanalyze aspects of the image data to determine relevant correlationsbetween historical ERCP procedures and the live ERCP procedure.Additionally, in some embodiments, available position data associatedwith the medical device may also be provided as input to the trainedmodel.

Responsive to determining, at step 710, one or more types of predictivenavigational guidance cannot be determined (e.g., due to lack ofnecessary information, an inability of a machine-learning model toidentify correlations between live procedure data and historicalprocedure data, etc.), an embodiment may, at step 715, transmit an alertnotification (e.g., to the user device 105). The alert notification maybe an audio notification, a visual notification, or a combinationthereof and may contain an explanation indicating why no dynamicguidance could be provided. Alternatively, in another embodiment, noadditional action may be taken.

Conversely to the foregoing, responsive to determining, at step 710, oneor more types of predictive navigational guidance, an embodiment may, atstep 720, generate one or more visual representations associated withthe determined predictive navigational guidance. In an embodiment, theone or more visual representations may correspond to one or more:annotations identifying relevant anatomical objects, trajectoryrecommendations for maneuvering the medical device and/or componentsthereof, and/or feedback notifications alerting a medical deviceoperator to updates occurring in the medical procedure.

At step 725, an embodiment may transmit instructions to a user device todisplay/overlay the visual representations of the predictive guidanceovertop some or all portions of the image data. For example, in anembodiment, the server system 115 may be configured to transmitinstructions to the user device to annotate one or more relevantanatomical objects during the medical procedure. In an embodiment,potential annotations may include anatomical objectcoloring/highlighting (e.g., where each detected relevant anatomicalobject is colored a different specific color, etc.), ROI designation(e.g., where relevant zones in the image data are delineated via atarget box or outline, etc.), text identifiers (e.g., where eachdetected relevant anatomical object is textually identified, etc.), acombination thereof, and the like. Turning now to FIG. 8 , anon-limiting example of annotations overlaid atop image data associatedwith a target papilla is provided. The annotations present in FIG. 8include visually distinguished anatomical objects (e.g., an intramuralsegment/Diverticulum 82 may be colored in blue, an intramural biliaryduct 84 may be colored in grey, a biliary duct orifice 86 may be coloredin green, and a pancreatic duct orifice 88 may be colored in red) aswell as ROI designations (e.g., a papilla identifier box 90 and anorifice region outline 92).

In another embodiment, the server system 115 may be configured totransmit instructions to the user device to provide a recommendedtrajectory to help position, align, and/or advance the ERCP scope andguidewire. In an embodiment, the recommended trajectory may be providedas an overlay atop the image data of the anatomical object(s). Forexample, with reference to FIG. 9 , a non-limiting exampleimplementation of how a trajectory overlay may be implemented in twodifferent procedural situations is provided. In FIG. 9 , a traditionalendoscopic view of two papillas, i.e., 905A and 910A, is provided. Eachpapilla contains a distinct papilla pattern that influences the orificelocation of the biliary and pancreatic duct. More particularly, thepositioning of the ducts and corresponding orifices may vary between thetwo papillas illustrated in 905A and 910A. These differences may berepresented in diagrams 905B and 910B. Specifically, it can be seen thatboth the biliary and pancreatic ducts are accessible via a singleorifice in the papilla illustrated in 905A, whereas the biliary andpancreatic ducts may be accessible via a dedicated orifice in thepapilla illustrated in 910A. Based on learned knowledge, the predictivenavigational guidance model may be able to identify the proper ductlocations for each papilla type and thereafter provide a recommendedtrajectory overlay for access to the target duct, as illustrated in 905Cand 910C. More particularly, with reference to 905C, despite the biliaryand pancreatic ducts being accessible through a single orifice, thetrajectory overlay may still provide, at 905C-1, an approach trajectoryto cannulate the biliary duct and/or, at 905C-2, an approach trajectoryto cannulate the pancreatic duct. Additionally, with reference to 910C,the predictive navigational guidance model may be able to differentiatebetween the two identified orifices and thereafter provide, at 910C-1,an approach trajectory to cannulate the biliary duct and/or, at 910C-2,an approach trajectory to cannulate the pancreatic duct.

In another embodiment, the server system 115 may be configured totransmit instructions to the user device to provide feedback to thephysician when the movement of the endoscope and/or the guidewire straysfrom the recommended trajectory. In an embodiment, the server system 115may be configured to provide feedback immediately (e.g., when departurefrom the recommended trajectory is initially detected) or,alternatively, when the degree of departure from the recommendedtrajectory exceeds a predetermined threshold. In an embodiment, thefeedback may manifest in one or more different forms. For example, thefeedback may manifest as a visual alert (e.g., a text alert, icon alert,animation alert, etc., presented on a display screen of a user device,etc.), an auditory alert (e.g., provided via one or more speakersassociated with the user device, etc.), a haptic alert (e.g., vibrationof the medical device via one or more actuators, etc.), or a combinationthereof. In an embodiment, the feedback may be presented only once, atpredetermined intervals (e.g., every 5 seconds, 10 seconds, etc.), orcontinuously. In an embodiment, the feedback may be instructive and maysuggest adjustments that the medical device operator may make to align aprojected approach path with the recommended trajectory.

In an embodiment, the server system 115 may be configured to nottransmit any predictive navigational guidance unless a confidence weightof the predictive navigational guidance model with respect to a targetanatomical object is greater than a predetermined threshold. Moreparticularly, a confidence weight held by the predictive navigationalguidance model for a particular anatomical object (e.g., a papilla) mayfirst be identified. The confidence weight may be based on, or reflectedby, the training the predictive navigational guidance model has had witha specific anatomical object (e.g., a specific type of papilla orpapilla pattern, etc.), wherein greater training may correspond tohigher confidence. An embodiment may then determine whether thisconfidence weight is greater than a predetermined confidence thresholdand, responsive to determining that it is not, may withhold transmittingpredictive guidance to a medical device operator.

Returning to FIG. 7 , at 730, an embodiment may optionally update theERCP database with data associated with the live medical procedure. Thisupdate data may subsequently be used to further train the predictivenavigational guidance model (e.g., using the processes previouslydescribed in FIGS. 2-3 ). In an embodiment, the types of data obtainedfrom the live medical procedure that may be utilized to update the ERCPdatabase may include one or more of: captured anatomical feature data,detected medical device position/movement data, medical procedureoutcome data (e.g., was overall cannulation successful or unsuccessful,which parts of the ERCP procedure were successful or unsuccessful, werethere any post-procedure complications such as pancreatitis, etc.),navigational guidance accuracy data (e.g., how accurate was anannotation for identifying an anatomical object, how accurate was arecommended trajectory, etc.), and the like. In one embodiment, theupdate to the ERCP database may be a single, batched update. Forexample, an embodiment may hold all of the captured data associated withthe medical procedure until it is complete. Thereafter, an embodimentmay transmit all data associated with the medical procedure to the ERCPdatabase. Alternatively, in another embodiment, the ERCP database may beupdated with medical procedure data continuously (e.g., as the data isaccumulated). For example, an embodiment may transmit image dataassociated with a target anatomical object to the ERCP databasesubstantially immediately when it is captured, an embodiment maycontinuously transmit movement data associated with a medical deviceduring the medical procedure to the ERCP database substantiallyimmediately once it is detected, etc.

FIG. 10 is a simplified functional block diagram of a computer 1100 thatmay be configured as a device for executing the methods of FIGS. 2-3,and 7 , according to exemplary embodiments of the present disclosure.For example, device 1000 may include a central processing unit (CPU)1020. CPU 1020 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 1020 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 1020 may be connectedto a data communication infrastructure 1010, for example, a bus, messagequeue, network, or multi-core message-passing scheme.

Device 1000 also may include a main memory 1040, for example, randomaccess memory (RAM), and also may include a secondary memory 1030.Secondary memory 1030, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc., which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 1030 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 1000. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 1000.

Device 1000 also may include a communications interface (“COM”) 1060.Communications interface 1060 allows software and data to be transferredbetween device 1000 and external devices. Communications interface 1060may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 1060 may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 1060.These signals may be provided to communications interface 1060 via acommunications path of device 1000, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 1000 alsomay include input and output ports 1050 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

The systems, apparatuses, devices, and methods disclosed herein aredescribed in detail by way of examples and with reference to thefigures. The examples discussed herein are examples only and areprovided to assist in the explanation of the apparatuses, devices,systems, and methods described herein. None of the features orcomponents shown in the drawings or discussed below should be taken asmandatory for any specific implementation of any of these apparatuses,devices, systems, or methods unless specifically designated asmandatory. For ease of reading and clarity, certain components, modules,or methods may be described solely in connection with a specific figure.In this disclosure, any identification of specific techniques,arrangements, etc. are either related to a specific example presented orare merely a general description of such a technique, arrangement, etc.Identifications of specific details or examples are not intended to be,and should not be, construed as mandatory or limiting unlessspecifically designated as such. Any failure to specifically describe acombination or sub-combination of components should not be understood asan indication that any combination or sub-combination is not possible.It will be appreciated that modifications to disclosed and describedexamples, arrangements, configurations, components, elements,apparatuses, devices, systems, methods, etc. can be made and may bedesired for a specific application. Also, for any methods described,regardless of whether the method is described in conjunction with a flowdiagram, it should be understood that unless otherwise specified bycontext, any explicit or implicit ordering of steps performed in theexecution of a method does not imply that those steps must be performedin the order presented but instead may be performed in a different orderor in parallel.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware. The term “software”is used expansively to include not only executable code, for examplemachine-executable or machine-interpretable instructions, but also datastructures, data stores and computing instructions stored in anysuitable electronic format, including firmware, and embedded software.The terms “information” and “data” are used expansively and includes awide variety of electronic information, including executable code;content such as text, video data, and audio data, among others; andvarious codes or flags. The terms “information,” “data,” and “content”are sometimes used interchangeably when permitted by context.

Program aspects of the technology may be thought of as “products” or“articles of manufacture” typically in the form of executable codeand/or associated data that is carried on or embodied in a type ofmachine-readable medium. “Storage” type media include any or all of thetangible memory of the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide non-transitory storage atany time for the software programming. All or portions of the softwaremay at times be communicated through the Internet or various othertelecommunication networks. Such communications, for example, may enableloading of the software from one computer or processor into another, forexample, from a management server or host computer of the mobilecommunication network into the computer platform of a server and/or froma server to the mobile device. Thus, another type of media that may bearthe software elements includes optical, electrical and electromagneticwaves, such as used across physical interfaces between local devices,through wired and optical landline networks and over various air-links.The physical elements that carry such waves, such as wired or wirelesslinks, optical links, or the like, also may be considered as mediabearing the software. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

While the disclosed methods, devices, and systems are described withexemplary reference to transmitting data, it should be appreciated thatthe disclosed embodiments may be applicable to any environment, such asa desktop or laptop computer, an automobile entertainment system, a homeentertainment system, etc. Also, the disclosed embodiments may beapplicable to any type of Internet protocol.

It should be appreciated that in the above description of exemplaryembodiments of the invention, various features of the invention aresometimes grouped together in a single embodiment, figure, ordescription thereof for the purpose of streamlining the disclosure andaiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claimed invention requires morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive aspects lie in less than allfeatures of a single foregoing disclosed embodiment. Thus, the claimsfollowing the Detailed Description are hereby expressly incorporatedinto this Detailed Description, with each claim standing on its own as aseparate embodiment of this invention.

Furthermore, while some embodiments described herein include some butnot other features included in other embodiments, combinations offeatures of different embodiments are meant to be within the scope ofthe invention, and form different embodiments, as would be understood bythose skilled in the art. For example, in the following claims, any ofthe claimed embodiments can be used in any combination.

Thus, while certain embodiments have been described, those skilled inthe art will recognize that other and further modifications may be madethereto without departing from the spirit of the invention, and it isintended to claim all such changes and modifications as falling withinthe scope of the invention. For example, functionality may be added ordeleted from the block diagrams and operations may be interchanged amongfunctional blocks. Steps may be added or deleted to methods describedwithin the scope of the present invention.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other implementations, which fallwithin the true spirit and scope of the present disclosure. Thus, to themaximum extent allowed by law, the scope of the present disclosure is tobe determined by the broadest permissible interpretation of thefollowing claims and their equivalents, and shall not be restricted orlimited by the foregoing detailed description. While variousimplementations of the disclosure have been described, it will beapparent to those of ordinary skill in the art that many moreimplementations are possible within the scope of the disclosure.Accordingly, the disclosure is not to be restricted except in light ofthe attached claims and their equivalents.

We claim:
 1. A computer-implemented method for generating navigationalguidance for a medical device within a body, the computer-implementedmethod comprising: receiving, at a computer server, image dataassociated with at least one anatomical object; determining, using aprocessor associated with the computer server and via application of atrained predictive navigational guidance model to the image data,navigational guidance for the medical device in relation to the at leastone anatomical object; generating, based on the determining, at leastone visual representation associated with the navigational guidance; andtransmitting, to a user device in network communication with thecomputer server, instructions to display the at least one visualrepresentation associated with the navigational guidance overtop of theimage data on a display screen of the user device.
 2. Thecomputer-implemented method of claim 1, wherein the medical device is anendoscope comprising an extendable guide wire.
 3. Thecomputer-implemented method of claim 1, wherein the at least oneanatomical object corresponds to one or more of: a papilla, an orifice,or an internal duct, and the navigational guidance includes a path forthe medical device for cannulation of the at least one anatomicalobject.
 4. The computer-implemented method of claim 1, wherein the imagedata is captured by at least one sensor associated with the medicaldevice and/or by at least one other imaging device.
 5. Thecomputer-implemented method of claim 4, wherein the at least one sensorassociated with the medical device comprises a camera sensor and whereinthe image data captured by the camera sensor comprises at least one of:shape data, orientation data, and/or appearance data of the at least oneanatomical object.
 6. The computer-implemented method of claim 4,wherein the at least one other imaging device comprises an X-ray deviceand/or an ultrasound device, and wherein the image data captured by theat least one other imaging device comprises anatomical structure data.7. The computer-implemented method of claim 1, further comprisingreceiving position data captured by one or more other sensors associatedwith the medical device, wherein the one or more other sensors compriseat least one of: an electromagnetic sensor, an accelerometer, agyroscope, a fiber optic sensor, an ultrasound transducer, a capacitiveposition sensor, or an inductive position sensor.
 8. Thecomputer-implemented method of claim 1, wherein the determining thenavigational guidance comprises identifying, using the trainedpredictive navigational guidance model, anatomical feature data from theimage data.
 9. The computer-implemented method of claim 8, wherein theidentifying the anatomical feature data comprises: identifying, within afirst target region of the image data, a first classification associatedwith a first anatomical object of the at least one anatomical object;identifying, from within a second target region bounded by the firsttarget region, a second classification associated with a secondanatomical object of the at least one anatomical object; detecting, fromwithin the second target region, a location of one or more thirdanatomical objects; and detecting one or more other anatomical objectsassociated with the first anatomical object.
 10. Thecomputer-implemented method of claim 1, wherein the determiningcomprises: identifying a confidence weight held by the trainedpredictive navigational guidance model for the at least one anatomicalobject; and determining whether that confidence weight is greater than apredetermined confidence threshold; wherein the generating of thenavigational guidance is only performed in response to determining thatthe confidence weight is greater than the predetermined confidencethreshold.
 11. The computer-implemented method of claim 1, wherein theat least one visual representation comprises one or more of: at leastone trajectory overlay, at least one annotation, and/or at least onefeedback notification.
 12. The computer-implemented method of claim 11,wherein the at least one trajectory overlay includes a visualindication, overlaid on top of an image of the at least one anatomicalobject, of a projected path to an access point of the at least oneanatomical object that a component of the medical device may follow tocannulate the at least one anatomical object.
 13. Thecomputer-implemented method of claim 12, further comprising: receiving,at the computer server, position data for the medical device; andidentifying, based on analysis of the received position data, deviationof the medical device from the projected path; wherein the generatingthe at least one feedback notification comprises generating responsiveto detecting that the deviation of the medical device from the projectedpath is greater than a predetermined amount.
 14. Thecomputer-implemented method of claim 11, wherein the at least oneannotation comprises one or more visual indications, overlaid on top ofan image of the at least one anatomical object, indicating predeterminedfeatures associated with the at least one anatomical object.
 15. Thecomputer-implemented method of claim 14, wherein the one or more visualindications comprise one or more of: a color indication, an outlineindication, and/or a text-based indication.
 16. A method of training apredictive navigational guidance model, the method comprising:receiving, from a database, a training dataset comprising historicalmedical procedure data associated with a plurality of completed medicalprocedures; extracting, from image data in the training dataset,anatomical feature data; extracting, from sensor data in the trainingdataset, medical device positioning data; extracting, from the trainingdataset, procedure outcome data; and utilizing the extracted anatomicalfeature data, the extracted medical device positioning data, and theextracted procedure outcome data to train the predictive navigationalguidance model.
 17. The method of claim 16, wherein the training datasetis annotated with identification data.
 18. The method of claim 16,wherein the extracting the anatomical feature data comprises:identifying, from within a first target region in the image data, aclassification associated with a first anatomical object; determining,from within a second target region bounded by the first target region inthe image data, an identity of a second anatomical object; detecting,from within the second target region, at least one location associatedwith one or more third anatomical objects; and detecting one or moreother anatomical objects associated with the first anatomical object.19. The method of claim 16, further comprising: identifying a newprocedural outcome; and updating the database with data associated withthe new procedural outcome.
 20. A computer system for generatingnavigational guidance for a medical device within a body, the computersystem comprising: at least one memory storing instructions; and atleast one processor configured to execute the instructions to performoperations comprising: receiving image data associated with at least oneanatomical object; determining, using the at least one processor and viaapplication of a trained predictive navigational guidance model to theimage data, navigational guidance for the medical device in relation tothe at least one anatomical object; generating, based on thedetermining, at least one visual representation associated with thenavigational guidance; and transmitting, to a user device, instructionsto display the at least one visual representation associated with thenavigational guidance overtop of the image data on a display screen ofthe user device.